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ABSTRACT 

In certain space-restricted environments, many of the benefits resulting from computer technology 
have bee. bZZZZLc of the size, weight, io.cvenie.ce, and lack of mobility assctated ™th emstrng 

computer interface devices. Accordingly, an effort to develop a highly rnn^m^ cd^mable co p 

disolav and control interface device, referred to as the Sensory Integrated Data Interface (SIDI) is 
underway. The system incorporates a micro-video display that provides data display and ££ 
liehtweifdit headset Software commands are implemented by conjunctive eye movement and voice 
commands of operator. In this initial prottgyping effort, various -ttf^c-shclT cm^htnu^ 
integrated into a desktop computer and with a customized menu-tree software « P p te »“°° 
feasibility and conceptual capabilities. When fully developed as a customized system the interface device will 
“ob^>handsW operation of portable computer equipment. It will thus of 

information technology applications into those restrictive environments, both military and mdustna^ that 
not yet taken advantage of the computer revolution. This effort is Phase I of Small Business ln " ov ^ e 
Research (SBIR) Topic #N90-331 sponsored by the Naval Undersea Warfare Center Division, Newport. 

The prime contractor is Foster-Miller, Inc. of Waltham, MA. 


INTRODUCTION 

During the last decade, there has been a rapid development of technologies that support wmputer 
miniaturization. These technologies have significantly reduced computer workstation *z* j«d, *£» 
time, improved processor speed, memory capacity, and software ut.lizat.omIt is 

that functional improvements in the interface devices for computers have been relatively few. With the 
exception of the computer mouse/trackball and new "pen-based" input devices, virtually ^“P u ^ r ^ te “ s 
still rely on large and cumbersome interfaces, namely CRT/LCD screen displays and keyboards. There has 
been relativelyhttle ergogenic improvement with these interfaces since their 1950s predecessors were 

introduced. 

Unfortunately, many of the benefits associated with computer technology have been foregone 
because of this lack of appropriate interface devices. This is particularly true for certain space-restricted 
environments such as aircraft, submarines, and spacecraft. The disadvantages imposed by size, we^rt, znd 
inconvenience of existing interfaces as well as the lack of mobility for the operators using them has resulted 
in the avoidance of "computer solutions. 

The development effort described below seeks to solve this problem by integrating several new "off- 
the-shelf technology products to produce a highly miniaturized, micro-video 

svstem which incorporates both interactive ocular tracking and voice control. The goal is to create a 
"wearable" interfacTmounted on a portable headset. Further development will lead to the ^implementation of 
highly miniaturized, portable alternatives to currently used cathode-ray tube/hquid crystal display au 
keyboard control devices. The advantages of computer technology will thus become more accessib e 
space restricted and operator-mobile applications and environments. 
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DISCUSSION 


A number of emerging technologies are now available which can provide significant improvement for 
the human-to-computer interface. Singularly, each one of these technologies has the ability to provide 
improvement, but significant synergy is gained by combining them together into a single, integrated system. 
The technologies applied in this effort are those of micro-video displays, ocular tracking systems, and 
computer voice recognition. Based upon a capability verses cost analysis of commercially available products, 
as described in reference (1), the component subsystems described below were selected for use in this 
project. Since the interface concept relies on human sensory interaction, the system is referred to as the 
Sensory Integrated Data Interface (SIDI) 

Once the component subsystems were selected, the effort consisted of assembling them into a 
functional interface. On an adjustable headset were mounted a micro-video display, an associated Infra-Red 
(IR) illumination source and video camera, and a microphone. The micro-video display presents computer 
data to the operator. Using eye-tracking hardware installed in a PC expansion slot, screen cursor movement 
is caused to follow eye movement. Software menu selections are then implemented by gaze-fixation r iming 
and voice command. An analogy to the "point and click" functionality of a computer mouse can be drawn. 
The line-of-sight gaze is used to "point" to a menu option and the voice command "clicks on it" or activates 
the selected item. 

Figure 1 illustrates how the components were assembled on an operator headset. Note that the 
display projector/eye-tracker is worn in a monocular arrangement in front of one eye. This allows the 
operator to access computer data while operating other equipment or performing other tasks. Note also that 
the use of the headset allows "hands-free" operation of the computer. Figure 2 illustrates a high level 
component integration scheme for the desktop computer. A customized software application was developed 
to demonstrate conceptual capabilities. As shown in Figure 3, it consists of a menu-tree listing of topics 
which the operator activates by gazing and, when ocular lock is achieved, initiates with a verbal command. 
Running underneath this application are the customized software device drivers for the system components. 


Micro- Video Display 


Data display for SIDI is provided by the "Private Eye" developed by Reflection Technology, Inc. 

The Private Eye uses a new proprietary technology which uniquely combines conventional semiconductor 
and optical techniques to create an image of a 12 inch monitor in a miniature package measuring only 1.1 
inches X 1.2 inches X 3.2 inches. It weighs less than two ounces. The display uses emissive elements to 
provide a high definition, high contrast, and fast responding image viewable in bright daylight. 

Worn in a monocular position in front of one eye, the unit interfaces with a PC computer via an 
interface card in an expansion slot. A virtual image of the computer CRT display is projected approximately 
18 inches in front of the viewer’s eye. The unit displays 720 X 280 pixels which can be formatted as 25 lines 
with 80 characters per line or can be used to show bit-mapped graphics. The monochrome image appears to 
float in space in front of the viewer with a quality and resolution matching that of a standard display. A lens 
system allows image focusing to accommodate viewing with or without eye glasses. 


Ocular Tracking System 

The subsystem chosen to provide line-of-sight ocular tracking for the SIDI was the Eye Slaved 
Pointing (ESP) System from ISCAN, Inc. It is available as a PC compatible, turnkey eye movement 
monitoring system that serves as a hybrid interface device. Terminate and Stay Resident (TSR) software 
developed for the ESP is designed to substitute an operator’s line-of-sight gaze for standard computer 
pointing devices. The system provides cursor positional data to software applications which permit or require 
a pointing device. 
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The eye-tracker functions by using a custom set of integrated circuits and tracking algorithms to lock 
on and track a point on the operator’s eye in real-time. This is accomplished by measuring and tracking a 
low-level Infra-Red (IR) light beam reflected from the eye surface. A miniature IR video camera is used to 
acquire the pupil and corneal images needed by the tracking algorithms. The operator views a senes of 
positional data points on a display to calibrate the tracking function for the display. From that port on, the 
tracking function converts eye movement into screen cursor movement. Since the display and eye-tracking 
camera are positionally coupled by the headset, movement of the head does not affect the tracking func 10 
Both the ESP and Private Eye are mounted so as to be adjustable on the headset and allow vana e 
positioning of the components for different operators. 


Voice Recognition Unit 

The Voice Recognition Unit (VRU) chosen to provide voice control for the SIDI was the Voice 
Master from Covox, Inc. The system consists of a control unit, speaker, microphone, and^software which can 
provide both speech recognition and generation capability for a computer using an available RS-232 po t. 

The operator trains the system to accept specific voice commands which implement menu options once t e 
ESP has achieved ocular lock on a menu option. 

VRUs allow a computer to translate any spoken language into accurate, intelligent commands. It 
does so by breaking down a spoken command into its frequency components and then comparing those 
components with those of pre-stored commands. Most language requirements for a computer can be carrie 
out with a relatively small vocabulary when combined with gaze-directed screen interaction. Within 10 - 15 
minutes, an experienced operator can fully train the voice unit to recognize the required sets of alpha- 
numeric commands/inputs. 

While voice input alone can be used to select items from a display menu, there are many situations 
where it does not perform well. For example, voice commands are poor for most interactions with graphic 
images, like pointing to a specific wire in a schematic or selecting a location to insert information in a text 
display. Even with text applications, voice interaction is difficult when selecting one item from a disp ay 
which has many similar items, like a stock inventory sheet or a page of text. Eye-directed interaction in 
conjunction with voice commands handles these situations quickly and naturally. 


Software 

A customized software application was developed to demonstrate functional and conceptual 
capabilities of the SIDI. The software consists of both the user application and the underlying TSR that 
handles hardware interrupts. As shown in Figure 3, the user application consists of a menu tree that allows 
the operator to access various types of documentation for submarine system and subsystem components. 

The operator makes a selection by gazing at a major topic item. When the SIDI achieves ocular lock-on, the 
selection flashes at the operator signifying that verbal commands can be initiated. Lock-on is achieved 
quickly enough so that the operator perceives that the system is operating concurrently with his thought 
processes. By giving various commands such as "ACCEPT" , "UP", DOWN , and BACK the operator c ^ n 
move around the menu to view desired data. The software was written in the C programming language with 
some in-line assembly code. 


SIDI APPLICATIONS 

There is a wide spectrum of applications for the SIDI or a SIDI derivative. They range from various 
consumer entertainment products to business applications to military documentation applications. Any 
current computer application that could benefit from operator mobility is a candidate SIDI application 
Figure 4 illustrates an application for a technician performing a maintenance routine on a cruise missile using 
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procedure documentation displayed by a SIDI system. A few of the many possible implementations that 
could be envisioned are as follows: 


Paperless technical manuals 

Equipment diagnostic/repair systems 

Personalized training systems 

Entertainment electronics 

Portable inventory management systems 

Procedural documentation in space-restricted areas 

Computer interface for the physically handicapped 


FUTURE WORK 

., , .J}* 60 ?** ^, pr °j ect was to Prototype existing commercial off-the-shelf hardware to demonstrate 

e feasibility of the SIDI concept. That goal was achieved. A functional prototype system was developed 
t at successfully allow hands-free operation of a computer. The next step, as described in reference (2), is 
to refine the concept by developing a more compact and optimized interface device that provides operator 
mobility, requires less hardware adjustment to accommodate different operators, and is compatible with 
commercial software applications. 1 

To maximize operator mobility, it is proposed that the interface be developed to serve as a remote 
terminal served by a computer base station. The user module will contain the minimum amount of 
e ectromcs required to drive the display, eye tracking camera, and microphone. The base station will provide 
all the intelligence for these components via a data link. The data link will be preferably an un-tethered 
ment ° ^ ^ ** Infra Red ( IR ) s P ac,al optical link, thereby providing the operator unrestricted move- 


. . . J° reduce ‘he amount of component adjustment for each individual who operates the interface, the 

headset hardware will be miniaturized and integrated into a single, easy-to-use, composite unit. The 
hardware will be optimized for K 


Ease of alignment to obtain the eye image 
Ease of calibration to the user’s eye 
Better tolerance to room lighting variations 
Better display resolution 

A separation of component functions to allow base station interaction 


Finally, to preclude the need for customized software applications, the output of the eye tracking 
system will be modified to reflect standard computer mouse and trackball conventions. This will allow the 
use of standard software interface environments. 


1 . 

2 . 
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FIGURE 1 
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FIGURE 2 
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